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ABSTRACT 


This thesis focuses on analyzing factors that affect a student’s time to train (TTT) 
as he/she completes the Joint Strike Fighters (JSF) pilot training syllabus at the Integrated 
Training Center (ITC) on Eglin Air Force Base. It identifies the most robust course of 
action (COA), provides accurate TTT estimates, and identifies a watch list of factors that 
have the most effect on TTT. The results of this thesis provide a basis and a justification 
for procurement and resource decisions for the JSF. They also provide a means to ensure 
a proper flow of pilots to operational commands, reduce unnecessary flights, and enable 
the ITC to complete its mission while remaining within monthly flight time and airframe 
limits. It can also help predict the impact of future resource decisions, and assist 


commanders in mitigating their operational effect. 


The primary recommendation for reducing TTT is to have students work 11 hours 
per day and have the workday start at or after 0900. Increasing the number of full motion 
simulator (FMS), while reducing TTT, is not justified. Although a limited resource, 
airspace is not a potential bottleneck in the system and any change, short of elimination, 
has very little influence on TTT. Flight refly rates and aircraft failure rates above 12% 


and 13% respectively significantly increase TTT and must be carefully monitored. 
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EXECUTIVE SUMMARY 


The United States’ current inventory of fighter/attack aircraft is up against 
airframe time limits in the not too distant future. The United States Navy estimates a 
peak shortfall of 100 to 200 aircraft by 2018 (O’ Rourke, 2009). A new, low-cost, highly 


capable, and advanced replacement needs to be produced. 


In the early 90s, the Joint Advance Strike Technology Program (JAST) was 
created to bring together and mature the required technologies to create a new strike 
fighter capable of meeting the requirements of the Navy, Marines, and Air Force 
(Bolkcom, 2007). In 1994, Congress mandated that the JAST program be merged with 
the Advance Short Takeoff Vertical Landing program and the Joint Strike Fighter (JSF) 


was born. 


Eventually, two manufacturers were selected to compete for the JSF contract. 
Boeing and Lockheed Martin began a 4-year-long concept and demonstration phase that 
included producing a flying prototype. The competition ended in October 2001, when 
Lockheed Martin won the competition with its X-35 prototype. The JSF is now known as 
the F-35. 


The F-35 is produced in three variants. The F-35A is used by the Air Force and is 
a conventional takeoff and landing (CTOL) aircraft. It is a replacement for the F-16 and 
A-10, and will complement the F-22 for the Air Force’s high-low mix (Gertler, 2010). 
The F-35B is used by the Marines and is short-takeoff-and-vertical-landing capable; a 
requirement for use on amphibious ships. It will replace the AV-8 Harriers and older 
generation F-18s for the Marine Corps (Gertler, 2010). The F-35C is used by the Navy; it 
has foldable wings and is aircraft-carrier-launch-and-recovery capable. The three 
variants have a 70% to 90% commonality in parts (Gertler, 2010). This is an inherent 
design philosophy to reduce manufacturing cost and serviceability, and made the project 


truly joint. 


Unlike the F-22 Raptor, the F-35 is designed not just to be joint, but also 


multinational. To date, at least eight other countries, ranging from Australia to Turkey, 
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are expected to be partners in the production and procurement process (Bolkcom, 2007). 
One thing all the nations will have in common is that their pilots and maintainers will all 
initially be trained in the United States. The Integrated Training Center (ITC) is the 
multiservice, multinational schoolhouse for training personnel to pilot and maintain the 
F-35. The first ITC is located on Eglin Air Force Base in the Florida Panhandle. More 
training centers will be built as additional squadrons come on line and more aircraft enter 
service. The ITC has all the resources necessary to train U.S. and allied partners in flying 
and maintaining the F-35. It consists of an academic and operational side. The 
classrooms, simulators, and computer-based trainers are located on the academic side of 
the ITC, while the aircraft, brief rooms, and maintenance facilities are located on the 
squadron side of the ITC. The Navy, Marine Corps, and Air Force have separate 


squadrons and instructor pilots, but share all the resources that make up the ITC. 


In order to procure the correct number of resources (simulators, classrooms, 
instructors, etc.) for the ITC, Lockheed Martin built a model to simulate the flow of 
students through the training syllabus. By adjusting parameter levels, along with a set of 
ground rules and assumptions, student throughput can be studied to see their effects on 
metrics such as time-to-train (TTT) and resource utilizations rates. These results are 
analyzed and used to justify the number and type of each resource needed to satisfy the 
metrics and meet any requirements as specified by the end user. The goal of the ITC is to 
provide fully trained pilots and maintainers to meet the security needs of the future, as 
stated in the Joint Contract Specification document for the JSF: “The JSF Air System 
shall provide throughput training capacity to meet both peacetime and wartime service 


requirements for pilots and maintainers” (Born, 2007, p. 8). 


The ITC model simulates day-to-day ground and flight operations at the ITC. The 
model starts collecting data when students and resources, such as aircraft and flight 
simulators, are inducted into the system and ends when the last student graduates. Each 
student has a syllabus that lists all classes, meetings, and required activities that need to 
be finished before that student can graduate. Upon graduation of the last student, the ITC 
model produces detailed reports of each student’s experience. The time they start and 
finish each event is recorded, as well and any delay and reason for delay of any syllabus 


XV 


event. Resource utilization rates are recorded and used as justification for procurement or 


as an explanation for bottlenecks in the system. 


The ITC model is designed to run in either “steady state” or “ramp up” mode. 
The ramp up mode is useful in predicting the impacts of different resource schedules. 
The resources that can be varied in the ramp-up mode are the number of full motion 
simulators (FMSs), training aircraft, and instructor pilots. These inputs are provided to 
the model in the form of a text file containing a column for the week and a column for the 
change in number of that resource. This is the mode that most accurately describes the 
ITC as it is stood up and is the one that is studied in this thesis. Figure | summarizes the 


process up to the analysis. 


ITC Resources 
Instructor Mix © Time-to-train 


Modsim III ¢ Student Reports 
Discrete ¢ Instructor Reports 

7 Event ¢ Total ITC flight hours 

Student Induction Simulation ¢ Resource Utilization 

Schedule Model ¢ Aircraft FH/MO 


¢ Instructor work hours 









Pilot & Maintenance 
Curriculum 


Figure 1. Integrated Training Center Overview Diagram (From: Born, 2007) 


The objective of this thesis is to provide the Marine Corps with a consolidated 
plan that maximizes their resources and produces pilots for the JSF as quickly as the 
existing constraints will allow. With JSF production delays and the imminent loss of 
legacy aircraft, the Marines will soon be facing a fighter gap (O’ Rourke, 2009). To keep 
readiness high, the Marine Corps must be able to depend on its organic air-to-air and 
air-to-ground capability in an increasingly complex threat environment. This all hinges 
on the deployment of operational squadrons in a timely manner, and the efficient and 


correct usage of resources at the ITC. 
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This thesis explains and demonstrates a method to develop a best COA, 
identifying the resources having the most effect on reducing TTT. It also identifies the 
resources, and the limits or levels of these resources, which have the most effect on TTT. 
The impact of aircraft upgrades, the availability of training airspace, and changes to the 


students’ induction schedule and their effects on TTT are analyzed. 


The primary motivation for this research is to help the Marine Fighter Attack 
Training Squadron (VMEFAT-501) to effectively and efficiently train the next generation 
of pilots and maintainers. Communication with the squadron was essential and ongoing 
throughout the research, particularly in the design of each experiment. Their input 
narrowed the focus of the research and helped identify the most important aspects of the 
problem for further investigation. They were also important in validating the results from 


the perspective of the final customer. 


The overall recommendation for reducing TTT is to have students work 11 hours 
per day and have the workday start at or after 0900. Increasing the number of FMS to 
five or six, while reducing TTT, is not justified. Likewise, increasing the hours the ITC 
remains open has no effect on reducing TTT and should be ignored. The current plan of 
having four aircraft reserved for training events, while the excess undergo upgrades, is 
the best and most robust COA to ensure a minimum TTT. The availability of airspace for 
training, short of eliminating this resource, did not present a bottleneck to training and did 


not change TTT. 


The results produced by the ITC are inherently optimistic due to known 
deficiencies in how the model treats weather, maintenance, other stochastic elements, and 
in the strict scheduling of student events (Dummar, 2011). The ITC model uses means 
rather than distribution for its random variables, this also gives optimistic results. As a 
consequence, any reductions in TTT gained for choosing the COAs recommended in this 
thesis are also optimistic. These findings do provide a relationship between TTT and the 
controllable and uncontrollable factors. It is up to the commander and training officers to 


monitor these factors and use the COAs to help mitigate their effects. 
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I. INTRODUCTION 


A. BACKGROUND 


The United States’ current inventory of fighter/attack aircraft will be up against 
airframe time limits in the not too distant future. The United States Navy estimates a 
peak shortfall of 100 to 200 aircraft by 2018 (O’ Rourke, 2009). A new, low-cost, highly 
capable, and advanced replacement needs to be produced. The research into the 
replacement of the current inventory of aging third and fourth generation aircraft has 
been ongoing through the 1980s and 1990s, with no airframe making it to the production 
line. The early stages of the search for a new aircraft had the Navy, Marines, and 
Air Force seeking their own replacement, with different aircraft tailored to specific needs. 
In the early 90s, the Joint Advance Strike Technology Program (JAST) was created to 
bring together and mature the required technologies to create a new strike fighter capable 
of meeting the requirements of the Navy, Marines, and Air Force (Bolkcom, 2007). In 
1994, Congress mandated that the JAST program be merged with the Advanced Short 
Takeoff Vertical Landing (ASTOVL) program and the JSF was born. ASTOVL was a 
project begun in 1983 by the Defense Advanced Research Agency to replace the aging 
AV-8 Harrier. It was designed to be a multiservice replacement aircraft and fit well into 


the framework of the JAST program. 


Eventually, two manufacturers were selected to compete for the JSF contract. 
Boeing and Lockheed Martin began a 4-year-long concept and demonstration phase that 
included producing a flying prototype. The competition ended in October 2001, when 
Lockheed Martin won the competition with its X-35 prototype. The JSF is now known as 
the F-35. 


The F-35 will be produced in three variants. The Air Force variant, the F-35A, is 
a conventional takeoff and landing (CTOL) aircraft. It is a replacement for the F-16 and 
A-10, and will complement the F-22 for the Air Force’s high-low mix (Gertler, 2010). 
The Marine variant, the F-35B, is_ short-takeoff-and-vertical-landing capable; a 


requirement for use on amphibious ships. It is the most radical design of the three 


variants and, thus far, the most prone to setback (Majumdar, 2011). It will replace the 
AV-8 Harriers and older generation F-18s for the Marine Corps (Gertler, 2010). The 
Navy variant is the F-35C, which has foldable wings and is aircraft-carrier-launch-and- 
recovery capable. It is physically much larger than the two previous versions, due mainly 
to the requirement for airframe reinforcements because of the additional stress of catapult 
launches and arrested landing. It also has a larger wing area, due to the low speed 
approach profile required for a carrier-arrested recovery. The three variants have a 70% 
to 90% commonality in parts (Gertler, 2010). This is an inherent design philosophy to 


reduce manufacturing costs and serviceability, and made the project truly joint. 


Unlike the F-22 Raptor, the F-35 is designed not just to be joint, but also 
multinational. To date, at least eight other countries, ranging from Australia to Turkey, 
are expected to be partners in the production and procurement process (Bolkcom, 2007). 
One thing all the nations will have in common is that their pilots and maintainers will all 
initially be trained in the United States. The Integrated Training Center (ITC) is the 
multiservice, multinational schoolhouse for training personnel to pilot and maintain the 
F-35. The first ITC is located on Eglin Air Force Base in the Florida Panhandle. More 
training centers will be built as additional squadrons come on line and more aircraft enter 
service. The ITC has all the resources necessary to train U.S. and allied partners in flying 
and maintaining the F-35. It consists of an academic and operational side. The 
classrooms, simulators, and computer-based trainers are located on the academic side of 
the ITC, while the aircraft, brief rooms, and maintenance facilities are located on the 
squadron side of the ITC. There will be three squadrons, one for each variant of the JSF. 
The Navy, Marine Corps, and Air Force have separate squadrons and instructor pilots, 


but share all the resources that make up the ITC. 


In order to procure the correct number of resources (simulators, classrooms, 
instructors, etc.) for the ITC, Lockheed Martin built a model to simulate the flow of 
students through the training syllabus. By adjusting parameter levels, along with a set of 
ground rules and assumptions, student throughput can be studied to see their effects on 
metrics such as time-to-train (TTT) and resource utilizations rates. These results are 
analyzed and used to justify the number and type of each resource needed to satisfy the 
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metrics and meet any requirements as specified by the end user. The goal of the ITC is to 
provide fully trained pilots and maintainers to meet the security needs of the future, as 
stated in the Joint Contract Specification document for the JSF: “The JSF Air System 
shall provide throughput training capacity to meet both peacetime and wartime service 


requirements for pilots and maintainers” (Born, 2007, p. 8). 


B. PURPOSE 


The objective of this thesis is to provide the Marine Corps with a consolidated 
method that maximizes their resources and produces pilots for the JSF as quickly as the 
existing constraints will allow. With JSF production delays and the imminent loss of 
legacy aircraft, the Marines will soon be facing a fighter gap (O’ Rourke, 2009). To keep 
readiness high, the Marine Corps must be able to depend on its organic air-to-air and air- 
to-ground capability in an increasingly complex threat environment. This all hinges on 
the deployment of operational squadrons in a timely manner, and the efficient and correct 
usage of resources at the ITC. To address these issues, this thesis explains and 
demonstrates a method to: 

® Develop a best Course of Action (COA), identifying the resources having 

the most effect on reducing TTT. 


® Identify key resources, and the limits or levels of these resources that have 
the most effect on TTT. 


6 Examine airspace issues in the model. Current analysis has no limits on 
the airspace available. Airspace used for training can, and often needs to 
be, scheduled in advance. Some airspace 1s affected by prevailing weather 
conditions and may not be available for all training flights equally. A 
constraint on the availability of this resource will increase TTT. 


® Analyze the impact of aircraft upgrades on the overall student TTT. 


The primary motivation for this research is to help Marine Fighter Attack 
Training Squadron 501 (VMFAT-501) to effectively and efficiently train the next 
generation of pilots and maintainers. Communication with the squadron was essential 
and ongoing throughout the research, particularly in the design of each experiment. Their 


input narrowed the focus of the research and helped identify the most important aspects 


of the problem for further investigation. They were also important in validating the 


results from the perspective of the final customer. 


1. Literature Review 


There is almost no documentation for the ITC model itself. The model is 
proprietary software developed for Lockheed Martin and not meant for public use. The 
source code is unavailable and any change to the model must be made through the Joint 
Program Office and is dependent on funding availability. Of note, the model has been 
through the verification, validation, and accreditation process. It has been tested on 
several cases using the F-16, F-18, AV-8B, and GR-7 aircraft and their syllabuses. Two 
prior analyses of the ITC simulation exist. The latest was completed by Joe Dummar for 
his 2011 master’s thesis, and the other by Marry McDonald, a Research Associate at the 
Naval Postgraduate School, in 2010. 


Dummar’s thesis shows that the weather model used by the ITC is wrong and 
gives inherently optimistic results. The ITC uses weather for all aircraft events. Weather 
is either good or bad, with a certain probability depending on the month. Each aircraft 
event has an independent roll of the dice to determine if weather is good and the flight 
can be completed. Dummar suggests changing the weather modeling in the ITC 
simulation to one that models weather using Markov chains and a weather resource, since 
weather conditions for consecutive events are not independent (Dummar, 2011). He 


finds that the improved approach significantly affects estimates of TTT. 


McDonald’s analysis shows the performance of the ITC model has very high 
variability between replication (McDonald, 2010). The longest delay for scheduled 
events resulted from “student match” and “night flight.”” Student match means a student 
had to wait for another student in order to complete an event requiring a minimum of two 
students. This is why all classes in the ITC are started in multiples of two students, so 
they can progress through the syllabus together. They stay synched together throughout 
their experience and, if one falls behind, the other student ends up waiting longer at an 
event queue requiring two or more pilots. This is a simulation limitation that can lead to 


longer training times for that class, but is an issue that can easily be resolved at the 
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squadron level. Night flight delay means a student has run out of time to complete the 
night event before the ITC closes and it gets rescheduled for the next day. This is another 
simulation limitation that may lengthen training duration, which the squadron can easily 


manage. 
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HW. DATA AND METHODOLOGY 


This chapter describes the ITC model’s behavior, 1.e., the inputs, outputs, and 
ground rules, and assumptions required to initiate a run. Descriptions of the student and 


resource induction files are also covered. 


A. METHODOLOGY 


The ITC model simulates day-to-day ground and flight operations at the ITC. The 
model starts collecting data when students and resources, such as aircraft and flight 
simulators are inducted into the system and ends when the last student graduates. Each 
student has a syllabus that lists all classes, meetings, and required activities that need to 
be finished before that student can graduate. It also lists the order in which they are 
taken. The model strictly enforces this order, with no exceptions. This aspect of the 
simulation is a departure from how training is actually completed at a squadron. This 
limitation is acknowledged by Lockheed Martin, who claims the problem is “not a critical 
one and may only tend to make the ITC throughput estimates conservative” (Born, 2007, 
p. 11). Upon graduation of the last student, the ITC model produces detailed reports of 
each student’s experience. The time they start and finish each event is recorded, as well 
and any delay and reason for delay of any syllabus event. Resource utilization rates are 
recorded and used as justification for procurement or as an explanation for bottlenecks in 


the system. 


The ITC model is a discrete-event, first-in first-out, queuing-based simulation. It 
is implemented in ModSim III version 2.1 (Kenney, 2009). Currently, all random 
behavior in the model is based on a uniform zero to one distribution. Each event in a 
syllabus depends on wait queues that holds students, instructors, and resources needed to 


complete the event. Figure 2 shows this process in more detail. 


Prior to a flight or simulator event, the students must prepare for the event at the 
mission planning station (MPS). This takes a fixed period of time, which is entered via 
the user at the start of the simulation in the graphical user interface (GUI). Once 
planning starts, that student and MPS are made unavailable until completion of the 


7 


planning, after which the MPS is returned to the MPS queue awaiting another user. The 
student then waits for an instructor to become available. Once that happens, the 
instructor is taken from his queue. The student briefs the event to the instructor for the 
time specified in the GUI; then both wait for a simulator to become available. Once a 
simulator is available, the mission is flown for a duration dependent on the student’s 
syllabus. The instructor and student are not released from the event until after a debrief, 
which, again, takes a fixed period of time and is defined in the GUI. At that point, the 
instructor is returned to his queue and the student will move to his next event in the 
syllabus with a probability of success defined in the GUI. This same sequence occurs for 


each simulator, aircraft, and mission trainer, or any computer-based training requirement. 


3. Full Mission Simulator (FMS) 


Note: Re-fly only allowed 
Once for each unique 
FMS event. 


Available Return MPS Available Return FMS 
¥ 15 minute setup for next mission 
MPS Q FMS Q 


Next Event 


t 


Fly Mission Re-ty 
Syllabus Duration Re-fly) 


Note: MPS = Mission Planning Station GUI = Graphical User Interface 


Available 
Instructor Q Return Instructor 


© 2010 Lociheed Martin Corporation 





Figure 2. Full Mission Simulator (FMS) Event Flow (From: (Gray, 2010) 


The ITC model is designed to run in either “steady state” or “ramp up” mode. 
The steady state mode is used once all the resources have arrived at the ITC and their 
numbers will remain relatively constant for the duration of the simulation’s time horizon. 


This mode is useful in predicting the impacts of maintenance stand-downs, long-term 
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runway closures, or any resource constraint that affects an already operational training 
squadron. The ramp up mode is useful in predicting the impacts of different resource 
arrival schedules. Depending on when the first students begin to arrive for training, not 
all the resources approved may be ready for use at the ITC. The resources that can be 
varied in the ramp up mode are the number of FMSs, training aircraft, and instructor 
pilots. These inputs are provided to the model in the form of a text file containing a 
column for the week and a column for the change in number of that resource. This is the 
mode that most accurately describes the ITC as it is stood up and 1s the one that 1s studied 


in this thesis. 
B. MODEL INPUT AND OUTPUT 


1. Resources 


The following is a list of resources accounted for in the ITC model (Kenney, 


2009, p. 7): 


® Training Devices 

e Classrooms 

e Self-paced Computer Based Training Work Stations 

6 Mission Brief/De-brief Rooms for completion of flight events 
brief/debrief 

e Military Instructor Pilots (IPs) 

@ Military Maintainer Instructors 

8 Contract academic instructor pilots 

e Training Aircraft 

8 Runways and training airspace (military operating areas, low level 


routes, scored target ranges). 


Figure 3 shows an example of the GUI. On this page, changes can be made to the 
ITC facility resources, the air space resource, and the weather cancelation rates. The user 
can also specify the location of the pilot syllabus file and indicate whether to induct 


aircraft and FMSs, or use the steady state number shown on the GUI. 


F-35 Program Information 
Non Export Controlled Information - Releasable to Foreign Persons 
Lockheed Martin Proprietary Information 


Facility/ Aircraft Resources Student Syllabus Inductions 
Unclassified Classrooms Pilot Syllabus [Multiple + | 
Classified Classrooms Pilot Inductions (USMC_COA4_Eglin.txt | 
ICW Work Stations 
Full Mission Simulator [Induct FMS [~ Maint. Skill Sets > 3 (Single Induction File) 


MRT Flight Simulator 


Mission Planning Station 


Eglin_ITC_Mx_12_MSSR_Sec6_ 20080122 txt 





TTT TT TAT 


Brief/Debrief Rooms Avionics Syllabus [Avionics_Initial_2.txt ’ | 
Aircraft (Single Seat) Avionics Inductions Zeto_Avionics.tx#t tis 
Aircraft (Two Seat) [ Induct Aircraft 
ASMT Mechanical Syllabus [Mechanical_Initial_2.txt ’ 
WLT Mechanical Inductions [Zero_M echanical txt ~ 
ESMT 
Engine Trainer Weapons Syllabus (Weapons_Initial_2.txt + 
OML Weapons Inductions [Zero_Weapons.txt v 
Note: If CFS = 0, will use FMS Copyright Lockheed Martin Corporation, October 2007 
Airspace Runway Availability 
Sortie weather cancellation rate (%] 
MDA Slots (30 
MOA/T arget Slot holds One Mission 
LowLevels [30 Oct-Feb [15 Mar-Sept [30 
Low Level Entry Spacing = 6 minutes Target Slots (30 ; ; ; 
Note: Simulation Starts in January 
Runways (5 


oK | Cancel | 


Figure 3. Integrated Training Center Graphical User Interface Page One 


Z. Inputs 


This is the collection of controlled and uncontrolled variables or factors to be 
adjusted in the design of experiments. The user can make inputs via a GUI or input text 


file (Kenney, 2009, p. 8). 


e Student Induction Schedule (text file). 

© Course Curriculum/Syllabus (text file). 

® Training Equipment and Aircraft Resources (GUI). 
@ Instructor Resources (GUI). 

e Airspace and Runway Resources (GUI). 


® Weather Cancellation Rates (GUI). 
@ Full Mission Simulator Planning, Brief, Debrief duration (GUI). 
@ Probability of Aircraft Maintenance (GUI). 


® Aircraft Repair Time (GUI). 
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® Aircraft Preparation Time for the Next Flight (GUI). 
® Simulator Training Event Re-fly Rates (GUI). 

e Aircraft Training Event Re-fly Rates (GUI). 

@ ITC Training Duration Days/Year (GUI). 

e ITC Training Duration Hours/Day (GUI). 


Within the input text file there are more parameters that can be varied. From the 
student induction file the number of students per class, student work day, and student 
start time can be changed. This is an exhaustive list of all factors the user can adjust. 


Not all of these factors are changed in every experiment. 


3. Input Text Files 


All of the experiments will use essentially the same induction file. Any 
modification of the default induction file for a particular experiment is explained in 


Chapter III, Design of Experiments. 


a. Student Induction File 


The student induction file tells the simulation what week a new class 
starts, how many students are in each class, what syllabus each class is assigned, the time 
their workday starts, and how long their workday is. The default induction file has 
60 classes, with two students starting in each class. The first students start in week 83 
and the last students start in week 230, a period of almost three years. The students all 
have a 0700 start time and a 9-hour workday. Each class is assigned one of 14 different 
syllabuses. Multiple classes can be inducted on the same week. Table 1 1s an example of 
the default student induction schedule for the first 18 classes. The actual schedule is a 


text file and does not contain any of the header information. 
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Table 1. Student Induction File 


Schedule Students Start Hour Length Day 
DCMA_Initial_ Cadre.txt 10 700 9 
B1.0 TX_ALL_B Eglin.txt 10 700 
DCMA_Initial_ Cadre.txt 10 700 

TX3_B1.0.txt 10 700 
TX4_B1.0.txt 10 700 
B1.0 TX3 A Eglin.txt 10 700 
B1.0 TX4 A Eglin.txt 10 700 
TX3_B1.0.txt 10 700 

B1.0 TX_ALL_B Eglin.txt 10 700 
TX4_B1.0.txt 700 
TX4_B1.0.txt 700 
TX3_B1.0.txt 700 

B1.0 TX_ALL B Eglin.txt 700 
TX3_B1.0.txt 700 
TX3_B1.0.txt 700 
TX4_B1.0.txt 700 
TX4_B1.0.txt 700 





9 
9 
9 
9 
9 
9 
9 
9 
9 
9 
9 
9 
9 
9 
9 
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b. Aircraft Induction File 


This file tells the simulation at what week to change the aircraft inventory. 
Aircraft are added or removed from the inventory at the beginning of the week. There 
can never be a negative inventory and there is no maximum. The first aircraft are 
expected to arrive in week 83, which corresponds to October 2011, and the last aircraft is 
to arrive in September 2012. After September 2012, or approximately week 127 of the 
ITC calendar, there will be a total of 12 aircraft at the ITC. This schedule is based on 
VMFAT-501’s fiscal year 2012 flight hour plan and is subject to change. For simulation 
purposes, the exact dates are not required as long as each experiment uses the same 


induction file. 


Cc. Instructor Pilot Induction File 


Instructor pilots are inducted at a slightly faster rate than the aircraft. 
Some of the instructors are removed to fill instructor roles at other squadrons. The end 


goal is to have the same number of instructors as training aircraft (Kenney, 2009). The 
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induction file contains the week an instructor is added or removed from the ITC. Any 
changes take place at the beginning of the week. The inventory cannot go negative and 
there is no limit as to how many instructors are in the ITC. Unlike other resources, 
having too many instructors can affect the TTT. All winged aviators, including IPs, are 
required to maintain a minimum number of annual flight hours for proficiency 
(Operations, 2009). Individual squadrons or wings may further restrict the requirements 
to monthly minimums. Having too many instructors could lead to a loss of flight hours 
available for the students and increase their TTT. This is because there are limits to 
cumulative flight hours per month and flight hours per aircraft. These limits are not 


modeled in the ITC model. However, the ITC model records the levels used. 


d. FMS Induction File 


Flight simulators are an essential part of the ITC and are introduced in a 
similar fashion as the aircraft. As of spring 2011, VMFAT-501 is expecting four FMSs 
(A. C. Liberman, personal communication, April 18, 2011). The default FMS file has 
four FMSs arriving according to a schedule provided by VMFAT-501’s training officer. 


4. Outputs 


After each run of the simulation, when the last student graduates, the following 


items are output by the model (Kenney, 2009, p. 8): 


e Student Time-to-Train. 


® Number of Students in training as a function of time (1.e., student 
on-board count). 


@ Resource Utilization Hours (classrooms, training devices, 
instructors, aircraft, ranges). 


@ Training Equipment Utilization Rates as a percentage of the 
training day for each month of the simulation run (this is also 
shown for each day of the simulation run for the FMS). 


e Aircraft Flight Hours/month. 


e Aircraft takeoff and landing times, including any delays 
experienced for the runway, airspace, and target ranges. 


® The training day and number of instances on that day that a 
classroom was not available when required. 
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e Instructor training hours per week. 


The output from all runs are merged to get a distribution of outcomes, which are 
analyzed to find important trends and decisions that minimize the student’s TTT or 


maximize a training resource. Figure 4 summarizes the process up to the analysis: 


ITC Resources 


Instructor Mix ¢ Time-to-train 


Modsim II! ¢ Student Reports 
Discrete ¢ Instructor Reports 


| Event ¢ Total ITC flight hours 
Simulation ¢ Resource Utilization 
schedule Model ¢ Aircraft FH/MO 
¢ Instructor work hours 
Pilot & Maintenance 


Figure 4. Integrated Training Center Overview Diagram (From: Born, 2007) 





C. GROUND RULES AND ASSUMPTIONS 


A basic understanding of the ground rules and assumption is important in 
understanding how the model attempts to imitate reality and where it can break down. It 
also helps in the design of experiments and how to interpret the results of the model. The 
following assumptions are verbatim from the “model accreditation support package” 
(Kenney, 2009, pp. 8-10). 

e Normal operating hours for the Integrated Training Center are 
0700-2300 Monday through Friday. Briefing rooms and 
equipment will be available to support a 0700 take off and 
commencement of simulator events at 0700. Simulation of a 
single shift operation can be accomplished by giving all students 


the same work day start time (1.e., 0800) with a work day length of 
9 hours (workday ends at 1700). 


® Students are inducted evenly throughout the year and commence 
training on Monday of their induction week. The induction files 
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can be adjusted to simulate a non-uniform flow of students into 
the ITC. 


More than one class may be inducted in the same week. 


The optimum pilot class size for JSF is six students. The current 
ITC classroom design for pilots can accommodate up to 12 
students per class. 


Pilot and Maintainer classes are service unique. 
The first flight brief for the day occurs at 0445 for a 0700 takeoff. 


The first flight simulator brief of the day occurs at 0600 for a 0700 
simulator event. 


The student workday is a 9.0 hour day (start to finish) and students 
are assigned a specific “start workday time.” The intention of the 
9.0 hour day is to give each student 8.0 hours per day when 
training can be accomplished and 1.0 hour per day for a meal 
break. Maintenance training for the U.S. Services is accomplished 
on two shifts. 


In no case will students and instructors be scheduled for a training 
event sooner than 12.0 hours following the completion of their last 
event from the previous day. As an example, a student and/or 
instructor who finishes a flight debrief at 2200 will not be 
scheduled for a training event before 1000 the next day. 


Maximum crew duty day for instructor pilots is 12.0 hours 
measured from the start of the first training event to engine 
shutdown. This ground rule is based on a USAF Instruction AFI 
11-202 Volume 3 General Flight Rules, “For single seat aircraft or 
when only one pilot has access to the flight controls, the maximum 
flight duty period is 12 hours.” The normal instructor work day is 
9.0 hours. 


A Full Mission Simulator event and flight event will be permitted 
on the same day for the same student if they can both be 
accomplished within the students 9.0 hour work day. 


Flight simulator set up time is 15 minutes for each training event 
(after initial daily startup). 


Students will be permitted to fly more than one flight event on the 
same day. This only occurs for short flights such as Field Carrier 
Landing Practice events due to work day length restrictions. 


Student planning time for each flight event is 1.0 hour utilizing the 
Mission Planning Station. 


Brief time for training flights 1s 2.25 hours prior to takeoff. 
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Flight debriefs end 3.0 hours after landing. 
Day flights land before 1900 and night flights takeoff after 1900. 


Airborne aggressor requirements are satisfied by Training 
Squadron JSF aircraft flown by Training Squadron instructor 
pilots. 


After landing, there is a 10% probability the aircraft will require 
repair. Time to repair is 3.0 hours followed by a 1.5 hour 
preparation time for the next flight event. These parameters are 
adjustable via the user interface. 


Aircraft turnaround time for the next flight event is 1.5 hours and 
is adjustable via the user interface. 


Cancellations for weather occur after student planning and prior to 
commencement of the flight brief. Weather cancellation rates are 
5% and are adjustable via the user interface. 


Re-fly rates for flights and simulator events are 8% and 5% 
respectively and are adjustable via the user interface. The re-fly 
decision is made at the end of debrief. 


Navy Carrier Qualification (CQ) re-fly rate 1s 12.5%. Students 
who repeat CQ will re-fly all simulator and flight events associated 
with the CQ phase of training. 


Students will not re-fly the same flight or simulator event more 
than once. 


Student planning time for each Full Mission Simulator event is 30 
minutes and is adjustable via the user interface. Student planning 
utilizes a Mission Planning Station. 


Full Mission Simulator brief time is 60 minutes and is adjustable 
via the user interface. 


Full Mission Simulator debrief time is 30 minutes and 1s adjustable 
via the user interface. 


The number of Military instructor pilots is equal to the number of 
training aircraft in accordance with direction received from the 
Services and the JSF Program Office. All airborne flight 
instruction is provided by Military instructor pilots. 


After completing a flight debrief, the instructor will not be 
scheduled for a subsequent flight brief for 1.0 hour. 


On each training day, there is a 2% probability that each instructor 
and student may be unavailable for a 2.0 hour time period. This is 
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intended to account for medical appointments and unexpected 
absences for both instructors and students. 


® Minimum allowable spacing between flights on the same low-level 
route is 10 minutes. 


® Landing aircraft have priority over departing aircraft for the 
runway. 
® There are a total of 246 days per year that are available for 


[training]. This parameter is selectable via the user interface. 


e Students will take a 15-minute break in between non-like training 
events (e.g. simulator to classroom, classroom to simulator, etc.). 
Breaks during successive classroom training events will be handled 
by the instructor. 


The ITC model repeatedly uses a point estimate rather than a random draw from a 
distribution for its events. For some events, such as maintenance repair times, subject 
matter experts expect significant variability. Using point estimates in lieu of probability 
distributions, especially in high-utilization situations, will likely result in overestimating 


the system’s throughput. 


These model assumptions are used in conjunction with recommendations from 
VMEAT-501 to help design and analyze the experiment. Their contribution cannot be 
summarized in table form, but is an important part of how the simulations are designed 
and what factors need to be investigated. This helps reduce the complexity of the data 


and reduce the number of COAs to explore. 


In Chapter III, these factors and assumptions are brought together to form an 
efficient design that can be used to get the most information from the model, while 
keeping within computational and time constraints. Emphasis is on what factors, at what 
levels, have the greatest effect on TTT and how VMFAT-S501 can use that information to 


mitigate any unforeseen circumstances. 
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Hil. DESIGN OF EXPERIMENTS 


This chapter discusses how the experiments in this thesis are designed and the 
reason behind those designs. There are three separate experiments intended to answer the 
questions posed in Chapter I. The first experiment analyzes the effect of aircraft 
upgrades and modifications. The second experiment analyzes airspace and its effect on 
the ITC model. The last experiment analyzes syllabus variables—those factors that can 
be changed in the student induction files—and their effects on TTT. The noise factors, 
signal factors, and the levels of each in the three separate experiments are explained in 


detail. 


The designs are run on a cluster of computers at the Naval Postgraduate School’s 
Simulation Experiments and Efficient Designs (SEED) Center in Monterey, California. 
Each design point is repeated 50 times with different random number seed values. The 
outputs of each run are concatenated together to form output files representing all the 
runs and repetitions. This output is merged with the original design and analyzed using 


the JMP statistical package (Sall, Creighton, & Lehman, 2005). 


A. WHY DESIGN EXPERIMENTS? 


There are many different factors and levels of those factors in the ITC model to 
consider, and only a limited amount of computer resources and time to complete the 
study, thus an efficient design of experiments 1s crucial. With a proper design, all aspects 
of a complex model can be studied with greater fidelity and exponentially less time than 
looking at all combinations of factors and levels. The majority of the experiments use a 
mixed design, based on crossing a traditional factorial design with a nearly orthogonal 
Latin hypercube (NOLH) design (Cioppa & Lucas, 2007). The NOLH 1s the basis for all 
continuous factors of concern, while categorical factors use the factorial design. The 
algorithm used for their generation can be found on the SEED Center website 


(http://harvest.nps.edu/). 
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1. Robustness 


The goal sometimes is not just finding a solution that maximizes or minimizes the 
metric of interest, but a solution that also looks at the variability of that optimal solution. 
A robust solution is one that may sacrifice mean performance to minimize variability. It 
is a solution that is stable around a larger variation of uncontrollable inputs with little 
change in output. This way of looking at and analyzing the data is useful when there is a 
lot of variability in the response, uncertainty in the inputs, and when the variance of the 
response is not constant for the entire range of the regressors. This condition applies to 


our situation. 


The inputs consist of signal and noise factors. The signal factors are inputs that 
the user has control over. The number of aircraft, instructor working hours, and students 
per class are all signal factors. The weather’s cancellation rate, aircraft repair times, and 
aircraft failure rates are all unknown and uncontrollable. They can best be described by a 
distribution, but the ITC simulation does not currently take probability distributions as 
input variables. Because the JSF is a new system, the distribution of failure rates and 
repair times are unavailable. The model uses point estimates for these and the other noise 
factors. Unfortunately, queuing theory tells us that this will likely bias the results 
(Ross, 2007). 


In robust analysis, the intent is to look at the effect of the signal factors over the 
entire distribution of noise factors. To do this, both signal and noise factors are expanded 
into separate orthogonal or nearly orthogonal designs covering the full range of their 
values. These two separate designs are then crossed to form a master design. Once the 
data are generated, it is collapsed over the noise variables. That is, for a given signal 
design point, the response is generated by averaging all of the design points in the noise 
matrix. This gives a distribution of outcomes for each design point of the signal factors. 
This is better than using a single value for the noise factors, but not as good as using a 
distribution. It gives equal weight to all possible values of the noise factor when they are 
probably not all equal, and can lead to higher variability in the outcome. The 
methodology used for the robust analysis in this thesis is further described in 


Sanchez (2000). 
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2. Factors and Levels 


The factors of concern, affecting TTT and their levels (or range of values), are 
dependent on the problem being studied. All factors come from the list of inputs shown 
in Chapter II or otherwise are explained separately. Not all of these factors are used in 
every experiment, nor will their ranges be the same. Each experiment has a specific goal, 
and thus the range and number of factors is adjusted to best match the need of the 
experiment. If the full values of each factors’ ranges were used they may dominate the 
factors of interest and hide the response. This is a very important consideration when 
varying noise factors in the design, especially when all possible levels have a probability 
of occurrence. For example, having aircraft cancellation rates near 100% will dominate 
the output and make it difficult to see the effects of the signal factors. This narrowing 
down of the correct range for the each factor is an iterative process involving subject 


matter experts. 


Unless otherwise stated in the experiment, the noise factors are at the default 


values used in historical analysis, shown in Table 2. 


Table 2. Noise Factors Default Levels 


MOA Slots __30__|FMS Refly % 
Weather cnx Summer % 


Target Slots 30 ~——s«| Weather cnx Winter % 
AC Fail Rate % 
Flight Refly % AC Repair Time (hours) 





B. AIRCRAFT UPGRADE AND MODIFICATION DESIGN MATRIX 


Once aircraft are delivered to the ITC they will be basic aircraft with minimal 
flight clearances and not yet fully mission capable. As testing continues, and the aircraft 
are certified for different flight profiles, they will be upgraded at the ITC. This upgrade 
may be a simple software replacement or a complex process where a flight control 
surface, hydraulic, electrical, or structural component is replaced. When these aircraft 


undergo an upgrade, they are unusable for the period of time required to complete the 
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upgrade. A question of interest to VMFAT-501 is what affect the upgrade process will 
have on TTT? 


Aircraft module upgrade is not an input to the system. The only way to simulate 
this as a factor is by using a perturbation of the original aircraft induction file. The 
aircraft delivery schedule is known. By modifying when aircraft arrive and taking them 
out of service when they undergo upgrades, multiple scenarios can be explored and 
compared to a base case of no upgrades. Depending on the complexity of the upgrade, 
the aircraft downtime could be from a few days to a few weeks. The current plan is to 
keep four aircraft available for training at all times, while the other aircraft go through the 
upgrade process (A. C. Liberman, personal communication, April 18, 2011). With this 


COA, no upgrade is made until there are more than four aircraft in the inventory. 


This perturbation of the aircraft induction schedule is based on the assumption 
that once the upgrade is made available, then all available aircraft—less those needed to 
complete training missions—will all undergo the upgrade at the same time and complete 
at the same time. The time needed to complete the upgrade depends on its complexity, 
but is assumed to be between one and three weeks for simulation purposes. If the 
upgrade is made available while there are only four aircraft in the inventory, none of 
those aircraft are upgraded until a new aircraft arrives. The upgrades are made as soon as 


the required minimum numbers of aircraft are present. 


The idea is to ensure a minimum of two aircraft at all times to cover the flight 
schedule, while the extra two are used as backups for any maintenance issues that may 
come up. This strategy of four to make two may not be the best choice for all situations. 
Changes in upgrade completion time or aircraft failure rates may favor a different COA. 
To understand which COA is best, the following rules are used to simulate all possible 
decision policies with respect to how many aircraft to hold in reserve. The number of 
aircraft held in reserve is between two and five, and is designated A, B, C, or D, 
respectively. The time to complete is one, two, or three weeks. Although it may take a 
few days, the simulation only accepts integer values representing the week to make a 
change to the aircraft inventory. Added to this is a control design, designated AO, where 


upgrades are not required or they take zero time to complete, corresponding to the 
pup 


original aircraft induction schedule. This makes a total of 13 different aircraft induction 
schedules or design points to explore. This design is crossed with a 10 factor (noise) 
NOLH design of 33 design points, forming a mixed design with a total of 429 runs to 
fully explore the problem. The individual upgrade scenarios are shown in Tables 3 and 4. 
The value under the “week” column represents when the aircraft inventory will change 
and the value under “amount” column represents the number of aircraft added to 


the inventory. 


Table 3. | Upgrade Design Signal Factors 


Aircraft to Keep |U 


re ee a 
a 





Table 4. | Upgrade Design Schedule Induction File 





weekt week2 week3 week4 weekS week6 week? week8 week9 amount! amount2 amount3 amount4 amountS amount6 amount? amount8 amount9 


ACO 83 87 90 98 118 121 126 127 128 3 2 2 { { 2 | 0 0 
ACt 83 86 88 91 99 119 122 127 129 2 { 2 2 { { 2 1 0 
AC2 83 89 92 100 120 123 128 129 130 2 3 2 { { 2 | 0 0 
AG3 83 87 89 90 92 101 121 124 129 2 { { { 2 { { 2 { 
Bi 83 89 90 91 99 119 122 127 128 3 | { a { { 2 | 0 
B2 83 90 91 92 100 118 120 123 128 3 2 { { { 0 1 2 { 
B3 83 93 96 101 121 124 129 130 131 3 3 1 { { 2 { 0 0 
C1 83 87 91 92 99 119 122 127 128 3 { 2 { { { 2 1 0 
(2 83 87 91 94 100 120 123 128 129 3 1 | c { { 2 { 0 
03 83 87 96 101 121 124 129 130 131 3 1 3 1 { 2 { 0 0 
DI 83 87 93 94 99 119 122 127 128 3 2 { { { { 2 1 0 
D2 83 87 96 98 100 120 123 128 129 3 2 1 { { { 2 1 0 
D3 83 87 98 99 121 124 129 130 131 3 2 { 2 { 2 { 0 0 


The noise factors used in this experiment are shown in Table 5. The minimum 
and maximum value of each factor is also shown. The refly values represent the 
probability that a student will have to refly an aircraft or FMS event. “Wx Summer” and 
“Wx Winter” represent weather cancellation rates for each flight in the summer or winter, 
respectively. “Fail Rate” and “Repair Time” represent the probability that an aircraft will 


break down before the flight and how long it will take to repair the aircraft. 
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Table 5. | Upgrade Noise Factors Nearly Orthogonal Latin Hypercube 
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The correlation of this design and its space-filling property are shown in Figure 5. 
Correlation is a measure of linear dependence between two factors and is a value between 
1 and -1. A zero correlation would mean the two factors have no linear dependence and 
is the goal in any designing. Correlation of 1 or —1 means a perfect positive or negative 
linear relationship between the two factors. The design is nearly orthogonal, with “FMS 
Refly” and “Runways” having the highest correlation of only 8%. This should not 
interfere with estimating the effects of different upgrade strategies. The scatterplot 
matrix shows the coverage achieved in the factor space with this design. Each point 
represents a specific run of the experiment. All pairs of signal and noise factors are 
plotted. 
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Multivariate 


Correlations 


Runways 1.0000 
MOA Slots 0.0381 
Low Level 0.0111 
Target Slots -0.0176 
Flight Refly -0.0694 
FMS Refly -0.0800 
Wx Summer 0.0199 
Wx Winter -0.0654 
AC Fail Rate -0.0331 
AC Repair Time 0.0150 
Scatterplot Matrix 


Runways 


0.0381 
1.0000 
0.0219 
-0.0310 
-0.0154 
0.0196 
-0.0038 
0.0043 
-0.0241 
0.0163 


0.0111 
0.0219 
1.0000 
-0.0018 
-0.0087 
-0.0151 
-0.0001 
0.0120 
-0.0158 
0.0761 


-0.0176 
-0.0310 
-0.0018 
1.0000 
0.0103 
-0.0500 
-0.0231 
-0.0049 
0.0048 
0.0345 


-0.0694 
-0.0154 
-0.0087 
0.0103 
1.0000 
-0.0173 
0.0056 
0.0347 
-0.0360 
0.0096 


-0.0800 

0.0196 
-0.0151 
-0.0500 
-0.0173 

1.0000 
-0.0057 
-0.0101 
-0.0000 

0.0695 


0.0199 
-0.0038 
-0.0001 
-0.0231 

0.0056 
-0.0057 

1.0000 
-0.0241 

0.0008 

0.0392 


-0.0654 
0.0043 
0.0120 

-0.0049 
0.0347 

-0.0101 

-0.0241 
1.0000 

-0.0155 

-0.0164 


-0.0331 
-0.0241 
-0.0158 
0.0048 
-0.0360 
-0.0000 
0.0008 
-0.0155 
1.0000 
0.0719 


Runways MOA Slots Low Level Target Slots Flight Refly FMS Refly Wx Summer Wx Winter AC Fail Rate AC Repair Time 


0.0150 
0.0163 
0.0761 
0.0345 
0.0096 
0.0695 
0.0392 
-0.0164 
0.0719 
1.0000 


2? Low Level 
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“== Wie Winter 
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Figure 5. Upgrade Design Multivariate Plot of Noise Factors 
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C. AIRSPACE EFFECTS ON TTT 


This experiment focuses on limiting the number of Military Operating Areas 
(MOAs), low-level routes, and target slots available for training. Each syllabus flight 
event uses one or more of these resources to complete its mission. An MOA is airspace 
reserved for military training and test flights. Low-level routes are routes in an MOA, or 
other special-use airspace, where military aircraft can train at speeds and altitudes that are 
otherwise prohibited under Federal Aviation Administration rules. Target slots or ranges 
are usually located in restricted airspace and are used by the military for live-fire 
exercises. All of these areas are shared with other military organizations and scheduled 
through a local control authority. The availability of these resources could then affect 
TTT in their own way. The ITC has queues for each of the airspace factors and 
depending on when and how many students need these resources there may be a delay in 
its availability. For this experiment, the noise and signal factors, and their levels, are 


Shown in Table 6. 


Table 6. Airspace Experiment Input Variables 


pf SIGNAR NSE 


2 


1 
2 
2 
4 
5 
4 
3 
3 
4 
5 
5 
4 
2 
1 
3 
3 
5 
2 
3 
4 
5 
2 
2 
5 
1 
4 
3 
3 
1 
4 
4 
2 





Each design is created using an NOLH. The final design is a cross of the two 
separate designs to form a composite design with 1,089 design points, which allows us to 
examine the relationship between airspace availability and TTT. The multivariate 
correlation plot of the final design is shown in Figure 6. The highest correlation between 


the signal factors is a very low 0.0053. 


Correlations 


MOA.Slots Low.Level Target.Slots Runways Flight.Refly FMS.ReflyWx.Summer Wx.Winter AC.Fail.Rate AC.Repair. Time 
MOA.Slots 1.0000 -0.0051 0.0001 0.0000 -0.0000 -0.0000 -0.0000 0.0000 -0.0000 -0.0000 
Low.Level -0.0051 1.0000 0.0053 -0.0000 -0.0000 0.0000 -0.0000 = -0.0000 -0.0000 0.0000 
Target.Slots 0.0001 0.0053 1.0000 0.0000 0.0000 -0.0000 -0.0000 = -0.0000 -0.0000 0.0000 
Runways 0.0000 -0.0000 0.0000 1.0000 -0.0157 ~~ -0.0691 0.0244 -0.0003 0.0019 0.0141 


Flight.Refly -0.0000 -0.0000 0.0000 -0.0157 1.0000 -0.0054 -0.0097 0.0004 0.0341 -0.0075 
FMS.Refly -0.0000 0.0000 -0.0000 -0.0691 -0.0054 1.0000 -0.0081 -0.0001 -0.0004 -0.0126 
Wx.Summer -0.0000 -0.0000 -0.0000 0.0244 -0.0097 -0.0081 1.0000 -0.0022 -0.0004 -0.0344 
Wx.Winter 0.0000 -0.0000 -0.0000 -0.0003 0.0004 -0.0001 -0.0022 1.0000 0.0053 0.0353 
AC.Fail.Rate -0.0000 -0.0000 -0.0000 0.0019 0.0341 -0.0004 -0.0004 0.0053 1.0000 0.0154 
AC.Repair.Time = -0.0000 0.0000 0.0000 0.0141 -0.0075 -0.0126 -0.0344 0.0353 0.0154 1.0000 





Figure 6. Airspace Design Multivariate Plot 


D. SYLLABUS VARIABLES 


This last experiment tests the effect of changing student start time, workday 
length, number of FMSs, and ITC work hours on TTT. The first two experiments vary 
each signal factor over the entire range of noise factors, but for this experiment, the noise 
factors are kept constant at their default values. By looking at only the signal factor, the 
degree of complexity of the design is reduced and allows a full-factorial design to be 
used. The signal factors are listed in Table 7, with information on their levels and 


variable types. 


Table 7. Syllabus Signal Factors 


Number of students in class 2, 6, 10 
Number of FMSs 


Continuous (hrs) 0600, 0900 
Length of workday Continuous (hrs) 
Schoolhouse hours Continuous (hrs) 12,19 





Increasing the class size directly affects TTT. As more students complete for a 


constant supply of resources, the availability of these resources become limiting and 


27 


bottlenecks are created. This results in more time spent waiting in queues and increases 
TTT. This factor was added to get a sense of how much that increase is and how the 
other signal variables affect TTT as student numbers increase. The students have a 
default start time and workday length of 0700 and nine hours, respectively. Increasing or 
decreasing the student start time allows more day or night flight events to be completed. 
Changing the workday length allows a student more time to schedule and complete 
events each day. Increasing schoolhouse hours allows more time for simulator and 
academic events to be scheduled in a single day. Increasing the number of FMSs should 
result in a decrease in the average TTT; but at what level of class size will this have the 


most pronounced effect? 


The goals for this experiment are screening the factors of interest, then learning 
what levels of those factors will minimize TTT. A full-factorial design is created with 
JPM software (Sall, Creighton, & Lehman, 2005). From the data, a meta-model of the 
system is created. Both linear regression and partition trees are used to explore the data. 
This design enables main effects and two-way interaction to be estimated with no 


confounding. Table 8 shows the design in tabular form. 
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Full-Factorial Design 


Table 8. 


Design Pt Start Time Stud Workday ITC Workday Numin Class Num FMS 


10 


12 
19 
19 
12 
19 
19 
19 
19 
12 
12 
12 
19 
12 
19 
19 
19 
19 
12 
12 
12 
12 
19 
12 
12 
19 
12 
19 
12 
12 
19 
12 
12 
12 
12 
19 
12 
19 
12 
12 
12 
19 
12 
19 
19 
19 
12 
12 
12 
19 
12 
12 
12 
19 
19 
19 
12 
12 
19 
19 
12 
12 
19 
12 
19 
19 
19 
19 
19 
12 
19 
19 
19 


11 


11 


10 


11 


11 
11 


10 
11 
12 
13 


10 


10 


11 
11 
11 


14 
15 


16 
17 


10 


11 


18 
19 
20 
21 


11 
11 
11 
11 
11 
11 


22 
23 


24 
25 


10 


26 
27 


28 
29 
30 
31 


10 
10 


11 


11 
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33 


10 
10 
10 
10 
10 
10 
10 


11 
11 


34 
35 


36 
37 


11 


38 
39 


41 


10 


11 


11 


42 


43 


10 


45 


11 


46 


47 


11 
11 


49 


50 
51 


52 
53 


11 
11 


10 


54 
55 


11 
11 


56 
57 


10 
10 


58 
59 
60 
61 


11 


10 


11 


62 


63 


65 


10 


11 
11 
11 
11 
11 


66 
67 


10 


68 
69 
70 
71 


10 


72 


29 


THIS PAGE INTENTIONALLY LEFT BLANK 


30 


IV. ANALYSIS 


This chapter analyzes the results of all three experiments and points to a preferred 
set of conditions and action that can be implemented to help reduce TTT. The analysis is 
performed using the JPM statistical package. The data are analyzed using partition trees 
and linear regression to explain the hierarchical relationship between the factors of 


interest and their effect on TTT. 


A. AIRCRAFT UPGRADE EFFECTS ON TTT 


The experiment to determine effects of aircraft upgrades on TTT is run twice. 
The first run had two students arriving with each class. This number of students, coupled 
with the induction schedule of IPs, aircraft, and FMS, did not stress the system enough to 
make any significant difference in TTT. With so few students onboard, the resources 
were adequate to cover the entire range of noise and signal variables. TTT remained 
fairly constant and did not vary by more than 10% from any run. The utilization rate for 
the FMS was less than 10% and the flight hours per month for the IPs were similarly 
around 10 hours per month, which is below the recommended proficiency minimum. 
This means IPs did not fly enough flight events to satisfy the required minimum flight 
hours and would be required to fly non-student-related flights. This is inefficient, since 
these extra flights are not flown in conjunction with student events. To solve this 
problem, a second experiment is run with the number of students per class set to six, 


using the same design matrix. 


The average utilization rate for FMS in this new design is 32%, while the average 
IP flight hours increased to 15 hours per month. For TTT, the result of this new design 
shows it 1s insensitive to which upgrade scenario is chosen. Table 9 shows how the mean 


TTT for all scenarios are statistically equal at a 95% confidence level. 
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Table 9. Analysis of Variance of Mean Time to Train of Upgrade Scenario 


Means for Oneway Anova 


Level Number Mean Std Error Lower95% Upper 95% 


A 99 208.469 1.6997 205.13 211.81 
AQ 33 206.703 2.9440 200.92 212.49 
B 99 208.624 1.6997 205.28 211.97 
C 99 208.342 1.6997 205.00 211.68 
D 99 208.373 1.6997 205.03 211.71 


Std Error uses a pooled estimate of error variance 


Because there is so little variability in mean TTT, a formal robust analysis of the 
result would prove fruitless and is not attempted for this experiment. However, it is 
interesting to note that the default scenario considered by VMFAT-501 of having four 
aircraft at all times for the flight schedule, mod C, does minimize the TTT and the 
standard error in all scenarios other than the control group, AO. Therefore, it is the most 


robust COA. 


Further analysis of the TTT data for this experiment shows a nonnormal 
distribution of the TTT (see Figure 7). There are two peaks in the distribution, a lower 
peak of approximately 190 days and a high peak of approximately 225 days. These peaks 


can be explained by looking at the partition tree of the data by the noise variables. 





Upgrade Experiment Histogram of Days to Train 


Figure 7. 


The majority of the cases with TTT in the left peak have refly rates less than 12% 
and four or more MOA slots. While the right peak 1s composed of cases where the refly 
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rate is equal to or greater than 12% and aircraft failure rates at or above 13%. These 
factors and levels should be closely monitored in order to anticipate and rectify 


possible delays. 


1. Partition Tree of Upgrade Data 


Another way to visualize the important factors in determining TTT is through a 
partition tree. This method of modeling splits the data into groups so as to minimize the 
total sum of squares error (SSE). The factor and level of the factor to split is chosen by 
the algorithm, based on which maximizes the reduction in the total SSE. In other words, 
the data is partitioned into two groups according to which factor and level explains the 
majority of the variation in the data. In JMP, the splits are controlled manually until the 
desired degree of accuracy is achieved or the resulting pool of data has reached a 
minimum required for further splits because of insufficient observations or insufficient 
variability in the response (Sall, Creighton, & Lehman, 2005). The partition of the 
upgrade data is shown in Figure 8. There are no signal factors present after three splits. 
It is not until split 26 that the choice of upgrade scenario becomes a factor in determining 


TTT. This further confirms that TTT is insensitive to the chosen upgrade COA. 


Number 
RSquare RMSE N_ of Splits AlCc 
0.631 10.974331 21450 3 163652 


All Rows 


Count 21450 LogWorth Difference 
Mean 208.31753 4320.6511 23.3793 
Std Dev 18.061806 


Flight Refly<12 Flight Refly>=12 


Count 9100 LogWorth Difference || Count 12350 LogWorth Difference 
Mean 194.85672 2149.9815 31.0717 || Mean 218.23602 1832.5486 19.0404 
Std Dev 11.915663 Std Dev 15.169222 


MOA Slots>=4 MOA Slots<4 AC Fail Rate<13 AC Fail Rate>=13 

Count 8450 || Count 650 Count 3900 || Count 8450 
Mean 192.63731 || Mean 223.70903 Mean 205.20836 || Mean 224.24879 
Std Dev 8.9045355 || Std Dev 7.7774442 Std Dev 14.29548 || Std Dev 11.292198 





Figure 8. __ Partition of Upgrade Data Including Signal and Noise Factors 
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By further expanding the partition tree, the upper and lower tails of the 
distribution can be explored. Design points with an average TTT near 181 days form the 
lower tail. They are composed of situations where flight refly is below 12%, MOA slots 
are greater than four, FMS refly is below 14%, winter cancellation rates are below 16%, 
and there are 11 or more target slots available each day. The upper tail is composed of 
situations with flight refly greater than 12%, aircraft failure greater than 13%, aircraft 
repair time greater than five hours, and less than 26 MOA slots. Looking at the partition 
tree with the noise data is not useful since these are things that cannot be changed. 
Looking at the partition only as a function of the signal factors will also be of little use 
since the means of each COA are so close. The difference in each COA 1s not enough to 
explain the variation in TTT. When it comes to what decisions to make that will help 
reduce TTT, the choice of what upgrade scenario to use should not be a factor to 


consider. 


B. AIRSPACE AND TTT 


Airspace is an important factor in the smooth operation of an ITC. Results from 
the upgrade experiment shows how important the number of MOA slots is in reducing 
TTT. The design described in Chapter III brings the other two elements of air space, 
target slots and low-level routes, into a full experiment to explore the impact of these 


resources on TTT. 


Results from this experiment are similar to those for the upgrade scenario. There 
is no relationship between airspace and TTT. This can be seen in a simple plot of TTT 


versus airspace resources (see Figure 9). 
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340 mean.TTT vs. MOA.Slots & 2 more mean.TTT 
—  mean.TTT 
—  mean.TTT 


—  mean.TTT 


mean.TTT 





0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30 
MOA.Slots Target.Slots Low.Level 


Figure 9. Airspace versus Time to Train Experiment | 


The plot is fairly stable, with the average TTT around 260 and with very little 
variation. The MOA plot shows a tendency toward higher TTT as the number of MOA 
slots decrease. This behavior is expected, but not present, in either of the other two plots. 
Instead, they are fairly constant except for two excursions above the mean at 20 and 25 
and 15 and 20 for the target and low-level plots, respectively. A linear regression of the 
data verifies what the plots show—there is no relationship between TTT and airspace 


(see Figure 10). 
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Analysis of Variance 


Sum of 

Source DF Squares Mean Square F Ratio 
Model 3 1031.64 343.879 0.7474 
Error 1085 49924164 460.131 Prob>F 
C. Total 1088 500273.28 0.5240 
Parameter Estimates 

Term Estimate Std Error tRatio Prob>|t| 
Intercept 261.6333 2.046956 127.82 <.0001* 
MOA.Slots -0.102315 0.072236 -1.42 0.1569 
Low.Level 0.0152619 0.072237 0.21 0.8327 
Target.Slots 0.0312555 0.072236 0.43 0.6653 

Figure 10. Airspace Regression Statistics 


The F statistic, the mean square of the model divided by the mean square of the 
error, shows the model is statistically insignificant. Mean TTT is a better predictor of 
actual TTT than any relationship between the number and type of airspace available. The 


parameter estimates show the factors have no significant effect on TTT. 


To verify these findings, and ensure that the | to 30 range in the airspace factors 
is not too large and thus masking a relationship with TTT, a second experiment is run. 
The parameters remain unchanged, and the design is an NOLH with 65 design points. 
The airspace factor levels in the second experiment are reduced by a factor of three. 
Airspace values can now range from | to 10, and should provide greater resolution on the 
effects, if any, of airspace on TTT. To reduce the amount of computation required, the 
number of repetitions is reduced to 35 at each design points. Figure 11 shows the linear 


regression of the second experiment. 


Analysis of Variance 


Figure 11. 


Sum of 

Source DF Squares Mean Square _ F Ratio 
Model 3 1027.380 342.460 0.8743 
Error 58 22719.563 391.717 Prob>F 
C. Total 61 23746.943 0.4598 
Parameter Estimates 

Term Estimate Std Error tRatio Prob>|t| 
Intercept 253.56757 9.5131 26.65 <.0001* 
MOA.Slots 0.5077986 0.940488 0.54 0.5913 
Low.Level 1.3412531 0.955644 1.40 0.1658 
Target.Slots -0.621882 0.953691 -0.65 0.5169 


Airspace Regression Reduced Levels Statistics 
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The results from both regressions confirm that airspace and TTT have no 
relationship in the present scenario. The best conclusions made from this experiment is 
that airspace is not a potential bottleneck in the system and any change short of 


elimination of the airspace will have very little influence on TTT. 


C. SYLLABUS VARIABLES AND TTT 


The data from this experiment are analyzed using both a linear regression model 
and a partition tree. The regression produces a good fit to the data, but does not explain 
or provide a set of rules or guidelines for the commander to use. Both courses of analysis 


do agree on what factors are most important. The partition tree is the best model because 


of its simplicity to view, understand, and explain. 


1. Partition Tree 


Figure 12 shows a partial regression tree to three levels. This tree explains the 
majority of the variability in the data, with a resulting R-squared over (0.99). The data 


used on the partition tree includes all 50 repetitions of the original 72 design points. 


All Rows 
Count 3600 LogWorth Difference 


Mean 355.66746 6660.64 520.144 
Std Dev 260.17873 


Num in Class(2, 6) 


2400 LogWorth Difference 1200 LogWorth Difference 
182.28614 36466.749 207.74 702.4301 146.57431 34.561 
Std Dev 104.62844 27.77679 


Num in Class(10) 


Num in Class(6) : Num FMS(4) 
LogWorth Difference || Count 1200 LogWorth Difference LogWorth Difference || Count 400 
78.416069 10321.955 20.2413 || Mean 286.15621 512.07516 22.9798 690.90975 65.34362 20.6956 |} Mean 725.47079 
Std Dev 10.256761 Std Dev 14.218747 Std Dev 20.211911 Std Dev 26.498016 


LogWorth Difference 
80.412553 36.2191 


Stud Workday>=11 || Stud Workday<11 Stud Workday>=11 

Count 600 || Count 600 Count 600 
Mean 68.295417 || Mean 88.536722 Mean 274.66629 
Std Dev 1.0783906 || Std Dev 2.0537228 StdDev 8.4991675 


Stud Workday<11 Start Time>=9 Start Time<9 

Count 600 Count 400 |} Count 400 
Mean 297.64613 Mean 680.56197 || Mean 701.25753 
Std Dev 8.2443801 Std Dev 14.835561 || Std Dev 19.577616 


Start Time>=9 Start Time<9 
Count 200 || Count 200 





Mean 707.36127 || Mean 743.58032 
Std Dev 17.736821]| Std Dev 20.833085 





Figure 12. Partition Tree of Syllabus Data 


The most influential factor affecting TTT is the number of students in each class. 
The mean TTT with 2, 6, or 10 students per class is 78, 286, and 702 days, respectively. 


Because of this, the analysis is broken down into three scenarios based on how many 


Students are in the classes. 


oy, 


Num in Class(2) 
Count 1200 LogWorth Difference 


Mean 78.416069 955.58346 20.2413 
Std Dev 10.256761 


Stud Workday>=11 Stud Workday<11 
Count 600 LogWorth Difference Count 600 
Mean = 68.295417 = 222.95834 1.9495 Mean 88.536722 


Std Dev 1.0783906 Std Dev 2.0537228 


LogWorth Difference 
286.07891 3.868 


Start Time>=9 Start Time<9 Start Time>=9 Start Time<9 

Count 300 LogWorth Difference || Count 300 LogWorth Difference} | Count 300 LogWorth Difference || Count 300 
Mean 67.320667 7.5623972 0.33275 || Mean 69.270167 9.3114532 0.3075] | Mean 86.602722 19.836533 
Std Dev 0.5003684 Std Dev 0.4156233 Std Dev 0.7240054 


LogWorth Difference 
0.76983 |) Mean 90.470722 22.17297 0.72392 
Std Dev 0.6478782 


Num FMS(6) Num FMS(5, 4) ; Num FMS(4) Num FMS(6) Num FMS(5, 4) 
100 200 Count 100 Count 100 200 
67.098833 67.431583 69.167667 || Mean 69.475167 Mean 86.0895 86.859333 90.229417 
Std Dev 0.4643068 || Std Dev 0.4814978 Std Dev 0.3921609]! Std Dev 0.3858885 Std Dev 0.6522727 


Num FMS(4) 

Count 100 
Mean 90.953333 
Std Dev 0.5359435]| Std Dev 0.5809016 





Figure 13. Partition Tree for Two Students per Class 


Scenario | is shown in Figure 13. If there are only two students per class, then it 
is recommended to have their workday set at 11 hours or more and start at 0900 or later. 
These factors, at these levels, would produce a TTT of approximately 67 days with a 
standard deviation (SD) of 0.5 days. Having student workdays of less than 11 hours, and 
starting before 0900, gives a mean TTT of 90 days with an SD of 0.6 days. It is 
interesting to note that if the number of FMSs could be increased for this group, it would 
only decrease TTT by no more than 0.5 days. This group is more sensitive to start times 


and workday lengths than to the number of FMSs, and adding additional FMSs would not 
help to significantly reduce the TTT. 
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Num in Class(6) 


Count 1200 LogWorth Difference 
Mean 286.15621 277.29454 22.9798 
Std Dev 14.218747 


Stud Workday>=11 


Count 600 LogWorth Difference 
Mean 274.66629 41.763571 9.33069 
Std Dev 8.4991675 


Stud Workday<11 


Count 600 LogWorth Difference 
Mean 297.64613 24.695279 7.12285 
Std Dev 8.2443801 


Num FMS(6, 5) Num FMS(4) 


Count 400 LogWorth Difference |} Count 200 LogWorth Difference| | Count 400 
Mean 271.55606 7.4364296 3.54272 || Mean 280.88675 2.8942124 3.86228 
Std Dev 6.5444958 Std Dev 8.5571345 


Num FMS(6, 5) 
LogWorth Difference 200 LogWorth Difference 


Mean 295.27185 1.9867978 1.88003 302.39469 0.5541158 1.21194 
Std Dev 7.3452021 Std Dev 7.9008957 


Start Time>=9 Start Time<9 Start Time>=9 Start Time<9 

Count 200 |} Count 200 Count 100 |} Count 100 
Mean 269.78469 || Mean 273.32742 Mean 278.95561 || Mean 282.81789 
Std Dev 5.9780346]| Std Dev 6.6205896 Std Dev 8.0041248]| Std Dev 8.6941816 


Num FMS(6) Num FMS(5) Start Time>=9 Start Time<9 

Count 200 Count 100 |} Count 100 
Mean 294.33183 : Mean 301.78872 || Mean 303.00067 
Std Dev 7.5228861 |) Std Dev 7.0572767 Std Dev 6.4180042 || Std Dev 9.140364 





Figure 14. Partition Tree for Six Students per Class 


Scenario 2 is shown in Figure 14. If there are six students, the best setting of 
factors 1s to increase the student workday to 11 or more hours and have them start at 0900 
or later. With four FMSs, this gives a TTT of 280 days with an SD of eight days. The 
effect of having more FMSs is now significant. Having five or more FMSs would 
decrease TTT by nine days, or about two weeks of training time. This benefit may justify 
the additional cost of more FMS resources. Not only does it decrease TTT, but the SD is 
also reduced, making it a robust decision. Another interesting outcome of more students 
is the increase in the variability of their TTT. The SD has gone from about 1%, with two 


students per class, to 4% of mean TTT, with six students per class. 
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1200 LogWorth Difference 
Mean 702.4301 146.57431 34.561 
Std Dev  27.77679 


Num FMS(6, 5) Num FMS(4) 

Count 800 LogWorth Difference Count 400 LogWorth Difference 
Mean  690.90975 65.34362 20.6956 Mean 725.47079 80.412553 36.2191 
Std Dev 20.211911 Std Dev 26.498016 




















Start Time<9 

200 LogWorth Difference |] Count 200 LogWorth Difference 
Mean 707.36127 1.9438919  6.26033||Mean 743.58032 1.4352359 6.13137 
Std Dev 17.736821 Std Dev 20.833085 


400 LogWorth Difference 400 LogWorth Difference 
Mean 680.56197 19.265273  12.4032||Mean 701.25753 14.129189 14.2655 
Std Dev 14.835561 Std Dev 19.577616 
































Num FMS(6) Num FMS(5) 

Count 200 || Count 200 
Mean  674.36037||Mean 686.76358 
Std Dev 12.793834 || Std Dev 14.152919 


Num FMS(6) Num FMS(5) 
Count 200 |} Count 200 
Mean  694.12477]|Mean 708.39028 
Std Dev 18.387548]| Std Dev 18.114261 


Stud Workday<11 || Stud Workday>=11 
Count 100 }} Count 100 
Mean 704.2311]]Mean  710.49143 
Std Dev 18.082674|| Std Dev 16.899422 


Stud Workday<11 || Stud Workday>=11 
Count 100} Count 100 
Mean  740.51463]] Mean 746.646 
Std Dev 20.928559}} Std Dev 20.382041 


Figure 15. Partition Tree for 10 Students per Class 


Any desire to further decrease TTT would have to focus on other resources that 
are limiting production. In this case, FMS, by itself, is not a bottleneck in the system. 


The next likely bottleneck may be from the number of IPs or training aircraft. 


Scenario 3 1s shown in Figure 15. If there are 10 or more students, the most 
important factor in reducing TTT is the number of FMSs. A reduction of 35 days is 


achieved by having five or more FMSs, along with reducing the SD by over six days. 


The next most important factor 1s increasing student start time. This is a change 
from the last two scenarios, where student workday was more important than student start 
time. Having students start at 0900 or later reduces TTT by 20 to 35 days, depending on 
the number of FMSs available. 


Of note is how little effect the student workday day has on the overall TTT. In 
scenarios one and two, it was the primary factor in reducing TTT. With 10 or more 
students, the advantages of longer workdays are almost insignificant. In one instance, 
having student workdays at or greater than 11 hours only reduces TTT by less than 0.2 


days. This is due to other constraints in the ITC such as instructors and/or classroom 
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availability. Even with longer workdays, the student is unable to utilize the limited 


resources. 


Clearly, having students take almost three years to complete training is 
unacceptable. No levels of the signal factor looked at in this experiment can decrease the 
TTT to an acceptable level. To further explore the best ways to decrease TTT when there 
are 10 or more students arriving with each class, this experiment needs to be expanded to 
look at other factors in the model. Possible candidate factors are number of aircraft, 
instructors, runways, and airspace available. Each of these factors will increase the TTT 
if they are in short supply, but collectively, as a group, their effects are magnified. 
Fortunately, this work is ongoing and research already begun to explore these additional 


factors as well as the inclusion of noise variables. 
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Vv. RECOMMENDATIONS 


A. BEST COURSE OF ACTION 


The following recommendations are made to help VMFAT-501 reduce TTT as 
they stand up the first JSF ITC at Eglin Air Force Base. These recommendations are 
based on analysis of three experiments over a finite time period with constrained 


resources, and are described in Chapters III and IV. 


The choice of upgrade scenarios and resulting loss of training aircraft from one to 
three weeks have very little effect on TTT when there are six or less students per class. 
The choice of using four aircraft to support flight operations, while excess aircraft 
undergo upgrades, is the most robust choice of the 12 scenarios looked at in the 
experiment, not including the control scenario. This is the current scenario VMFAT-501 
intends to implement. Factors to watch that could lead to increasing TTT are having 
flight and FMS refly rates above 12% and 14%, respectively, and winter cancellation and 
aircraft failure rates above 16% and 13%, respectively. All factors that have a significant 
effect on TTT are uncontrollable, and must be continuously monitored in order to have 


reasonable estimates and to anticipate changes to a student’s TTT. 


Airspace was also another factor that had little effect on TTT. The availability of 
airspace for training, short of eliminating this resource, did not present a bottleneck to 


training and did not change TTT. 


The most profound recommendations come from the last experiment. The 
recommendations all depend on the number of students per class. As student numbers 
increase from 2 to 10 students per class, so does the TTT. Increasing the class size 
directly affects TTT if the resources are kept constant. As the number of students 
increases, the magnitude of the effects also increases and the hierarchy of significant 
factors changes. The factors do not have the same weight for class sizes of 2, 6, or 


10 personnel. 


Start time was significant in all three cases, as determined by the partition tree in 
Chapter III. The commander’s best decision, regardless of the number of students, is to 
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have students start at or after 0900, to reduce the average TTT. This reduces TTT by 2, 
4, and 20 days for class sizes of 2, 6, and 10 personnel, respectively. This may be due to 
bottlenecks in the system for the schedule and completion of night events. The ITC is 
very strict and inflexible when it schedules students for events, which is very different 


from how a squadron runs its flight schedule. 


Student workday length was significant in only two of the cases tested. With 10 
or more students, the length of a student’s workday had no significant effect on reducing 
the TTT. With six or less it was the dominate factor in determining the TTT, reducing it 
by more than 20 days in both cases. A commander’s best decision is to have student 


workdays of 11 hours or more, if there are six or less students per class. 


FMSs are an expensive resources and their effect on TTT must be significant 
enough to justify their increased expense. The number of FMSs was significant in only 
two of the cases tested. With two students per class, increasing the number of FMSs 
beyond four has no significant contribution to decreasing TTT. With six students per 
class, the TTT only decreased by nine days, or almost two training weeks at the ITC, 
which is a significant amount of time. When compared to an average TTT of over a year, 
the two weeks may not be significant enough to justify more FMSs; rather, there may be 
other constraints that are more economically feasible and easier to overcome. When 
there are 10 students per class, the reduction in TTT is greater than 30 days, which is 
significant, but compared to the two plus years of TTT, it is not enough to justify more 
FMSs. In this case, the increase in TTT 1s due to constraints in other resources. Potential 
sources of this constraint are the number of IPs, aircraft, classrooms, or runways available 


for training. Interestingly, ITC operational hours have no effect on the TTT. 


The overall recommendation for reducing TTT is to have students work 11 hours 
per day and have the workday start at or after 0900. Increasing the number of FMS to 
five or six, while reducing TTT, is not justified. Likewise, increasing the hours that the 
ITC remains open has no effect on reducing TTT and should be ignored. The current plan 
of having four aircraft reserved for training events, while the excess undergo upgrades, 1s 


the best and most robust COA to ensure a minimum TTT. 
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The results produced by the ITC are inherently optimistic due to deficiencies in 
how the model treats weather, maintenance, other stochastic elements, and in the strict 
scheduling of student events (Dummar, 2011). The ITC model also uses means rather 
than distribution for its random variables, this also gives optimistic results. As a 
consequence, any reductions in TTT gained for choosing the COAs recommended in this 
thesis are also optimistic. These findings do provide a relationship between TTT and the 
controllable and uncontrollable factors. It is up to the commander and training officers to 


monitor these factors and use the COAs to help mitigate their effects. 


B. FURTHER STUDY 


The metric of focus has been TTT, but there are other factors that could also be 
included to build a more dynamic model of the process. Flight hours per aircraft and per 
IP are very important and highly correlated to the TTT. The JSF, like any other machine, 
will have mechanical, maintenance, and administrative limits driven by utilization that 
cannot be breached. Once the squadron receives its first aircraft, it will also receive an 
allotment of funds for fuel purchases each quarter. This allotment and the current cost of 
fuel will determine a flight-hour budget and a maximum total number of flight hours per 
month that the squadron can fly. This factor is not simulated in the ITC, but is recorded 
as part of the output. There are no limitations placed on how many hours or sorties an 
aircraft can fly, as there are limits on how many hours a student or instructor can work in 
a day. To capture this effect, an experiment would have to be designed to look at aircraft 
utilization for each month, coupled with a predicted flight-hour budget, to see which 
scenarios are fiscally possible and to help predict what level of funding to request for 
training. This can also help predict a more reliable estimate of TTT by ruling out 


infeasible scenarios. 


An investigation of other resources, not explored in this thesis, may be helpful in 
discovering other bottlenecks in the system. A study of the delays in completing events 
could help pinpoint a general direction in which to find and fix these bottlenecks. The 


explanations of these delays are often vague and may require multiple designs to properly 
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investigate them. Combining designs of experiment one, two, and three into one grand 


design may produce a better understanding of the relationship between all factors. 


This thesis looked at the ITC model during the ramp-up mode. Further studies in 
the steady state mode would be of value, especially once reliable statistics become 
available with respect to aircraft failure rates and repair times, as well as student attrition 
rates and weather cancellation rates. Without actual data on JSF training times, the ITC 
model can continue to be a reliable predictor and tool for the commander when faced 
with unforeseen changes in resources or unexplained changes in some of the noise 


factors. 
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